在这项工作中，我们将神经头部的头像技术推向百万像素分辨率，同时着重于跨驾驶合成的特别挑战性的任务，即，当驾驶图像的外观与动画源图像大不相同时。我们提出了一组新的神经体系结构和训练方法，这些方法可以利用中分辨率的视频数据和高分辨率图像数据，以达到所需的渲染图像质量和对新视图和运动的概括。我们证明，建议的架构和方法产生令人信服的高分辨率神经化身，在跨驾驶场景中表现优于竞争对手。最后，我们展示了如何将受过训练的高分辨率神经化身模型蒸馏成一个轻量级的学生模型，该模型是实时运行的，并将神经化身的身份锁定到数十个预定的源图像。实时操作和身份锁对于许多实际应用头像系统至关重要。

我们介绍了一个现实的单发网眼的人体头像创作的系统，即简称罗马。使用一张照片，我们的模型估计了特定于人的头部网格和相关的神经纹理，该神经纹理编码局部光度和几何细节。最终的化身是操纵的，可以使用神经网络进行渲染，该神经网络与野外视频数据集上的网格和纹理估计器一起训练。在实验中，我们观察到我们的系统在头部几何恢复和渲染质量方面都具有竞争性的性能，尤其是对于跨人的重新制定。请参阅结果https://samsunglabs.github.io/rome/

The task of video prediction and generation is known to be notoriously difficult, with the research in this area largely limited to short-term predictions. Though plagued with noise and stochasticity, videos consist of features that are organised in a spatiotemporal hierarchy, different features possessing different temporal dynamics. In this paper, we introduce Dynamic Latent Hierarchy (DLH) -- a deep hierarchical latent model that represents videos as a hierarchy of latent states that evolve over separate and fluid timescales. Each latent state is a mixture distribution with two components, representing the immediate past and the predicted future, causing the model to learn transitions only between sufficiently dissimilar states, while clustering temporally persistent states closer together. Using this unique property, DLH naturally discovers the spatiotemporal structure of a dataset and learns disentangled representations across its hierarchy. We hypothesise that this simplifies the task of modeling temporal dynamics of a video, improves the learning of long-term dependencies, and reduces error accumulation. As evidence, we demonstrate that DLH outperforms state-of-the-art benchmarks in video prediction, is able to better represent stochasticity, as well as to dynamically adjust its hierarchical and temporal structure. Our paper shows, among other things, how progress in representation learning can translate into progress in prediction tasks.

Recently, many causal estimators for Conditional Average Treatment Effect (CATE) and instrumental variable (IV) problems have been published and open sourced, allowing to estimate granular impact of both randomized treatments (such as A/B tests) and of user choices on the outcomes of interest. However, the practical application of such models has ben hampered by the lack of a valid way to score the performance of such models out of sample, in order to select the best one for a given application. We address that gap by proposing novel scoring approaches for both the CATE case and an important subset of instrumental variable problems, namely those where the instrumental variable is customer acces to a product feature, and the treatment is the customer's choice to use that feature. Being able to score model performance out of sample allows us to apply hyperparameter optimization methods to causal model selection and tuning. We implement that in an open source package that relies on DoWhy and EconML libraries for implementation of causal inference models (and also includes a Transformed Outcome model implementation), and on FLAML for hyperparameter optimization and for component models used in the causal models. We demonstrate on synthetic data that optimizing the proposed scores is a reliable method for choosing the model and its hyperparameter values, whose estimates are close to the true impact, in the randomized CATE and IV cases. Further, we provide examles of applying these methods to real customer data from Wise.

Compact and accurate representations of 3D shapes are central to many perception and robotics tasks. State-of-the-art learning-based methods can reconstruct single objects but scale poorly to large datasets. We present a novel recursive implicit representation to efficiently and accurately encode large datasets of complex 3D shapes by recursively traversing an implicit octree in latent space. Our implicit Recursive Octree Auto-Decoder (ROAD) learns a hierarchically structured latent space enabling state-of-the-art reconstruction results at a compression ratio above 99%. We also propose an efficient curriculum learning scheme that naturally exploits the coarse-to-fine properties of the underlying octree spatial representation. We explore the scaling law relating latent space dimension, dataset size, and reconstruction accuracy, showing that increasing the latent space dimension is enough to scale to large shape datasets. Finally, we show that our learned latent space encodes a coarse-to-fine hierarchical structure yielding reusable latents across different levels of details, and we provide qualitative evidence of generalization to novel shapes outside the training set.

The role of mobile cameras increased dramatically over the past few years, leading to more and more research in automatic image quality enhancement and RAW photo processing. In this Mobile AI challenge, the target was to develop an efficient end-to-end AI-based image signal processing (ISP) pipeline replacing the standard mobile ISPs that can run on modern smartphone GPUs using TensorFlow Lite. The participants were provided with a large-scale Fujifilm UltraISP dataset consisting of thousands of paired photos captured with a normal mobile camera sensor and a professional 102MP medium-format FujiFilm GFX100 camera. The runtime of the resulting models was evaluated on the Snapdragon's 8 Gen 1 GPU that provides excellent acceleration results for the majority of common deep learning ops. The proposed solutions are compatible with all recent mobile GPUs, being able to process Full HD photos in less than 20-50 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

This paper studies audio-visual suppression for egocentric videos -- where the speaker is not captured in the video. Instead, potential noise sources are visible on screen with the camera emulating the off-screen speaker's view of the outside world. This setting is different from prior work in audio-visual speech enhancement that relies on lip and facial visuals. In this paper, we first demonstrate that egocentric visual information is helpful for noise suppression. We compare object recognition and action classification based visual feature extractors, and investigate methods to align audio and visual representations. Then, we examine different fusion strategies for the aligned features, and locations within the noise suppression model to incorporate visual information. Experiments demonstrate that visual features are most helpful when used to generate additive correction masks. Finally, in order to ensure that the visual features are discriminative with respect to different noise types, we introduce a multi-task learning framework that jointly optimizes audio-visual noise suppression and video based acoustic event detection. This proposed multi-task framework outperforms the audio only baseline on all metrics, including a 0.16 PESQ improvement. Extensive ablations reveal the improved performance of the proposed model with multiple active distractors, over all noise types and across different SNRs.

从示范中学习（LFD）方法使最终用户能够通过演示所需的行为来教机器人新任务，从而使对机器人技术的访问民主化。但是，当前的LFD框架无法快速适应异质的人类示范，也无法在无处不在的机器人技术应用中进行大规模部署。在本文中，我们提出了一个新型的LFD框架，快速的终身自适应逆增强学习（FLAIR）。我们的方法（1）利用策略来构建政策混合物，以快速适应新的示范，从而快速最终用户个性化； （2）提炼跨示范的常识，实现准确的任务推断； （3）仅在终身部署中需要扩展其模型，并保持一套简洁的原型策略，这些策略可以通过政策混合物近似所有行为。我们从经验上验证了能力可以实现适应能力（即机器人适应异质性，特定用户特定的任务偏好），效率（即机器人实现样本适应性）和可伸缩性（即，模型都会与示范范围增长，同时保持高性能）。 Flair超过了三个连续控制任务的基准测试，其政策收益的平均提高了57％，使用策略混合物进行示范建模所需的次数少78％。最后，我们在现实机器人乒乓球任务中展示了Flair的成功。

知识图（kg）用于广泛的应用中。由于行业的数据量和多样性，KG生成的自动化是非常需要的。 KG生成的一种重要方法是将原始数据映射到给定的KG模式，即域本体论，并根据本体论构建实体和属性。但是，这种本体的自动生成是苛刻的，现有的解决方案通常并不令人满意。一个重要的挑战是在本体工程的两个原则之间进行权衡：知识方向和数据取向。前者规定，本体应该对领域的一般知识进行建模，而后者则强调反映数据特异性以确保良好的可用性。我们通过我们的本体研究方法重塑方法来应对这一挑战，该方法将给定领域本体论转换为较小的本体论的过程是自动化的，该本体学是KG模式。域本体论可以设计为以知识为导向，而KG模式涵盖了数据特异性。此外，我们的方法允许在循环中将用户偏好包含在内。我们证明了我们正在进行的有关本体研究重塑的研究，并使用实际的工业数据进行了评估，并有令人鼓舞的结果。

我们提出了一种从一个或几种视图中重建人头的纹理3D网眼的方法。由于如此少的重建​​缺乏约束，因此需要先验知识，这很难强加于传统的3D重建算法。在这项工作中，我们依靠最近引入的3D表示$ \ unicode {x2013} $ neural隐式函数$ \ unicode {x2013} $，它基于神经网络，允许自然地从数据中学习有关人类头的先验，并且直接转换为纹理网格。也就是说，我们扩展了Neus（一种最新的神经隐式函数公式），以同时代表类的多个对象（在我们的情况下）。潜在的神经网架构旨在学习这些物体之间的共同点，并概括地看不见。我们的模型仅在一百个智能手机视频上进行培训，不需要任何扫描的3D数据。之后，该模型可以以良好的效果以几种镜头或一次性模式适合新颖的头。